1. The Field of the Invention
This invention relates generally to a virtual set system, and more particularly to a method and apparatus by which a composite image is generated from a system of multiple cameras each having a frame buffer associated therewith to store virtual image data for producing a composite image based on the associated camera's field of view image data and the virtual image data from the frame buffer.
2. The Background Art
Calculating virtual sets is known in the art of computer generated imaging. A virtual set combines scenes viewed and recorded by a camera with scene elements (such as backdrops) from some other source. The scenes recorded by the camera are typically either moving or stationary objects and the scenes are photographed in front of a plain, colored wall commonly referred to as a "blue stage." Typically, the blue stage is colored blue or green. Special equipment or processes are then used to combine the camera scenes with the other scene elements. The combination is usually accomplished using compositing techniques.
For example, a typical application of compositing techniques involves combining a camera scene with a background scene, wherein the camera scene is a live person in the foreground and the background scene is an image or photograph of some other scene (a "synthetic scene"). Using these techniques, everything in the camera scene which is the color of the blue stage (e.g., blue) is replaced with the synthetic scene. Preferably, the combined image will then appear as though the person is located in the background scene.
Rather than just placing the camera scene in front of a two-dimensional background scene (but clearly not blending into the background scene) often it is desirable to make the camera scene appear as if it was immersed within a three-dimensional (3D) environment. In such cases, it is necessary to know, with considerable precision, enough information about the camera to generate the scene that the camera is "seeing" if the background scene were really part of the camera's environment. The background scene is generated with a 3D graphic engine or computer, often referred to as an image generator. Traditionally, an image generator includes a frame buffer that is refreshed with data every cycle. With each refresh, the above calculations require that the image generator know what a camera is "seeing" so that the image generator can create the corresponding scene from a virtual database. The required camera information includes knowing the XYZ position, the orientation (pan, tilt, and roll), field of view (zoom), and focus.
Virtual set systems are typically described as either three-dimensional (3D) virtual set systems or two-dimensional (2D) virtual set systems. A 3D virtual set system relies upon the use of 3D image generators for the real time rendering of 3D imagery for each camera viewport. Real time is defined to be 30 frames per second or better. The advantage of 3D virtual sets is that both the camera and scene elements can be moved freely within the virtual world. This freedom in movement provides the director with maximum flexibility creating video products. However, this freedom comes at a significant cost.
A 3D image generator for use in a 3D virtual set system is typically a very expensive and complex system. This is because rendering complex scenes in three dimensions is computationally demanding. In most 3D virtual set systems, the cost of the image generator represents the majority of the costs of the overall virtual set system. In addition, due to the complex nature of 3D image generation systems, these systems tend to have a higher failure rate than any of the other components of a virtual studio.
A 3D virtual set system using multi-camera configurations typically requires an image generator for each camera. This practice is very expensive and virtual set vendors have attempted to defray the costs of these additional generators by using lower quality, more cost effective graphics systems for the rendering of scenes from preview channels. Preview channels are systems which allow "good enough" graphics for preview use, but are unable to render graphics good enough for actual air time, i.e., a broadcast image. Although preview channels reduce the cost of multi-camera virtual set systems, the use of low cost preview channels introduces a need for some sort of upstream switching to ensure that the correct video image is composited with the correct background imagery. This additional level of switching introduces an additional level of complexity to the overall system design. In those cases when the director needs more than one source of high quality imagery (i.e., fades, dissolves, iso-channels) he is forced to use multiple sources of high quality imagery, which usually equates to multiple image generation systems.
For example, in fades, an image generator is required for each camera, thus, two image generators are required because two cameras are required for the fade. A fade requires two cameras because a first image slowly fades into a second image. The first image requires a high quality virtual image and the second image that is being faded into also requires a high quality virtual image. Both images appear concurrently during parts of the fade. If the second image is using only a preview channel, the fade cannot be carried out properly because the second image is only good enough for preview. Thus, the prior art requires a second image generator.
Contemporary 3D virtual set systems have numerous problems due to the real time nature of 3D virtual set systems. Because the image generation systems are required to generate a complete scene in a single frame time or better, many limitations must be imposed upon the virtual set system. For example, the virtual set databases are strictly limited in terms of the number of surface areas in the scene that must be supplied with an image and the amount of texture in the scenes. These surface areas to be supplied with images are commonly referred to as "polygons" by those skilled in the art. In addition, real time graphics algorithms cannot take advantage of advanced rendering techniques such as ray tracing or radiosity because the complexity of the calculations requires to much processor time. The result of these limitations are scenes that lack realism and the scenes are often described as "cartoon" like.
The 2D virtual set systems are less complex than 3D systems and do not suffer from many of the problems of the 3D systems. Although most of the above problems do not exist in a 2D virtual set system, the 2D virtual set system has other problems that make contemporary systems undesirable. In many video productions, a 2D virtual set system relies upon cameras that are typically fixed in a particular location and allow only pan, tilt, and in some cases zoom and focus. By restricting the movement of the camera, the virtual set designer is able to take advantage of cost effective, high quality image rendering techniques such as ray tracing and radiosity. Prior to a video production, the director can specify the planned location of each of the cameras and the set designer can then pre-render the imagery for each of the cameras.
However, significant limitations of 2D virtual set systems are (i) the impracticality of having 3D objects move in the image and (ii) the impossibility of having free moving cameras. In addition, 2D systems force the director to stay within the pre-determined constraints of each of the cameras pre-rendered scenes. A director typically does not have the flexibility during production time to re-position any of the cameras.
Of current interest is a virtual set system that takes advantage of many of the attributes of traditional 2D virtual set systems, but, in terms of performance, also incorporates a wide range of the attributes of 3D image generation systems.